Motion blur is a known problem in the field of digital photography which occurs due to unsteady hand movement or when objects are imaged which move with respect to the camera. In case of moving objects, the image is smeared along the direction of relative motion of the object.
Several deblurring techniques are known in the art to deblur a blurred image. One of these techniques is known as short exposure time imaging. Short exposure time imaging, however, raises several challenges. First of all, in case of short exposure time imaging, each individual captured frame is noisy, since lowering the exposure time lowers also the number of photons that reach the photo sensor. Furthermore, since not enough light is captured by the photo sensor, the color tone is also lost.
Modern cameras address the problem of motion blur with an image stabilization technique, where motion sensors control mechanical actuators that shift the sensor or camera lens in real time during the image exposure to compensate for the motion of the camera. However, this approach only compensates for motion blur which occurs due to the camera motion but does not compensate for blurring artefacts which occur due to the motion of the imaged object.
Other deblurring techniques used in conventional cameras are known which make use of a deconvolution of the captured image. However, with these approaches high spatial frequencies are lost because of the box-shaped nature of the camera exposure. This often results in smearing of the high frequency contents. The Fourier spectrum of conventionally used Point Spread Functions (PSF) contains zeros in its spectrum so that the inverse filtering will amplify noise and produce ringing artefacts, thus making the deconvolution an ill-posed problem.
The Flutter Shutter approach proposed by R. RASKAR et al., in “Coded exposure photography: Motion-deblurring using fluttered shutter”, published in ACM Trans. Graph., 25 (3): 795-804, 2006 makes the deconvolution problem well-posed due to its broadband filter behaviour. With this approach, the shutter is opening and closing over the exposure time according to a random binary coded sequence. The sequence is chosen such that the resulting motion blur PSF has a flat frequency spectrum and high spatial frequencies are preserved. The Flutter Shutter approach thereby modifies the line segment kernel (typical motion-blur kernel) to achieve a more broadband frequency response, which allows for dramatically improved deconvolution results. This approach, however, relies on user interaction for estimating the correct PSF, since the length and direction of the PSF depends on the object's motion. However, different objects or regions within the captured image may have different motion directions and velocities. Therefore, this technique provides rather good results for imaged objects moving with a known constant velocity. On the other hand, the Flutter Shutter approach has shown inadequate for imaging objects moving with unknown or varying velocity.
A further approach called Motion Invariant Photography (MIP) is disclosed in US 2009/0244300 A1 and A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman “Motion-Invariant Photography”, ACM Trans. Graph., 27(3): 1-9, 2008. The MIP approach addresses the above-mentioned challenges by mechanically moving the camera or the sensor or lens element during exposure time in such a way that the static and moving parts of the scene likewise become uniformly blurred within a certain range of velocities. With a known kernel, the blur can be removed with a single deconvolution, without the need of any motion estimation and image segmentation. However, the direction of the moving object must still be known. This kind of special mechanical motion of the camera makes the blur invariant of the object's velocity.
By analyzing motion blur as integration in the space-time domain, it is proven that the only integration curve that results in a motion-invariant PSF is a parabola. Therefore, in the MIP approach the camera is mechanically moved according to a one-dimensional parabolic function over time using a special hardware construction. In other words, the movement of the camera follows a parabolic path by moving laterally, initially at a maximum speed and slowing to a stop, and then moving in an opposite direction laterally, increasing in speed to a maximum speed of the range in the opposite direction and finally stopping. During imaging, the mechanically moved camera therefore blurs the entire scene. This blurring is in a manner which is invariant to the velocity of the moving parts within the scene. Since the entire scene, including the static and moving parts, is blurred with an identical PSF, the blur can be removed via a single deconvolution.
However, also the MIP approach shows several disadvantages. First of all, a complicated hardware arrangement is required in order to mechanically rotate the camera (or lens or sensor) in a parabolic fashion. Furthermore, it is not possible, or at least rather complex, to mechanically move the camera, respectively the camera sensor, in a linear fashion, for example, along a V-shaped trajectory. The MIP approach is also limited to a one-dimensional camera motion. Due to the restriction of the one-dimensional camera motion, the direction of the moving parts within the scene must be known in advance before moving the camera along its trajectory, which is again rather disadvantageous.